It's an operator used in expressions for easier duplication of an object, overriding some of it's public properties/fields (optional) with expression - MSDN

Currently it can only be used with records. But maybe there will be no such restriction in the future (assumption).

Here's an example how it can be used:

// Declaring a record with a public property and a private field
record WithOperatorTest
{
    private int _myPrivateField;

    public int MyProperty { get; set; }

    public void SetMyPrivateField(int a = 5)
    {
        _myPrivateField = a;
    }
}

Now let's see how with operator can be used:

var firstInstance = new WithOperatorTest
{
    MyProperty = 10
};
firstInstance.SetMyPrivateField(11);
var copiedInstance = firstInstance with { };
// now "copiedInstance" also has "MyProperty" set to 10 and "_myPrivateField" set to 11.

var thirdCopiedInstance = copiedInstance with { MyProperty = 100 };
// now "thirdCopiedInstance " also has "MyProperty" set to 100 and "_myPrivateField" set to 11.

thirdCopiedInstance.SetMyPrivateField(-1);
// now "thirdCopiedInstance " also has "MyProperty" set to 100 and "_myPrivateField" set to -1.

NOTE for reference types from MSDN:

In the case of a reference-type member, only the reference to a member instance is copied when an operand is copied. Both the copy and original operand have access to the same reference-type instance.

That logic can be modified by modifying the copy constructor of a record type. Quote from MSDN:

By default, the copy constructor is implicit, that is, compiler-generated. If you need to customize the record copy semantics, explicitly declare a copy constructor with the desired behavior.

protected WithOperatorTest(WithOperatorTest original)
{
   // Logic to copy reference types with new reference
}

And in terms of what benefits it gives, I think it should be quite obvious now, that it makes copying of instances much easier and convenient.

Basically, the with operator will create a new object instance (records only, for now), by "coping values" from the "source" object and override some named properties in the destination object.

For example, instead of doing this:

var person = new Person("John", "Doe")
{
    MiddleName = "Patrick"
};
 
var modifiedPerson = new Person(person.FirstName, person.LastName)
{
    MiddleName = "William"
};

you can do this:

var modifiedPerson = person with
{
    MiddleName = "Patrick"
};

Basically, you will write less code.

Use this source to get more details on the example above and official documentation for more examples.

Short answer is the following: with keyword in C# was added for easier copy of complicated objects, with a possibility to override some of the public properties. Examples are already briefly provided in the accepted answer.